(19) 



J 



Europalsch^ ptentamt 
European Patent Office 
Office europeen dee brevets 




(12) 



(43) Date of publication: 

09.06.1999 Bulletin 1999/23 

(21) Application number: 98304941.2 

(22) Date ot filing: 23.06.1998 



01) EP 0 921 375 A1 

EUROPEAN PATENT APPLICATION 

(51) Intel A G01C 11/02 



(84) Designated Contracting States: 

AT BE CH CY DE DK ES Fl FR GB GR IE IT LI LU 
MCNLPTSE 

Designated Extension States: 
AL LT LV MK RO SI 

(30) Priority: 03.12.1997 JP 33328797 

(71 ) Applicant: Mixed Reality Systems Laboratory Inc. 
Yokohama-shi, Kanagawa-ken (JP) 



(72) Inventors: 

• Endo, Takaaki, 

c/o Mixed Reality Systems Lab. Inc. 
Yokohama-shi, Kanagawa-ken (JP) 

• Katayama, Akihiro, Mixed Reality Syst. Lab. Inc. 
Yokohama-shi, Kanagawa-ken (JP) 

(74) Representative: Beresford, Keith Denis Lewis 
BERESFORD & Co. 
2-5 Warwick Court 
High Holborn 
London WC1R 5DJ (GB) 



(54) Camera layout for acquiring images 

(57) An image recording apparatus for recording im- 
ages sensed by at least two cameras attached to a ve- 
hicle, has a first camera which is arranged to have an 
image sensing direction agreeing with a first direction of 
a vehicle, a second camera which is arranged at a po- 
sition separated a known distance r from the first camera 



to have an image sensing direction agreeing with a sec- 
onddirection about 180° different from the first direction, 
and a unit for, when the vehicle travels in the first direc- 
tion, recording first image data sensed by the first cam- 
era and second image data sensed by the second cam- 
era a time duration later corresponding to the known dis- 
tance, in association with each other. 
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Description 

BACKGROUND OF THE INVENTION 

s [0001] The present invention relates to an image recording method and apparatus for describing a virtual space on 
the basis of sensed images and, more particularly, to a method and apparatus for processing sensed images from a 
plurality of cameras which are separated from each other as if they were obtained from cameras which are not separated 
from each other. The present invention also relates to a recording medium that stores a program for implementing the 
method. 

10 [0002] In recent years, attempts have been made to build civic environments where many people socially live in 
cyber spaces formed by computer networks. Normally such virtual spaces are described and displayed using conven- 
tional CG techniques. However, since CG expressions based on geographic models have limitations, Image-Based 
Rendering (IBR) based on sensed images has come to the forefront of the technology. 

[0003] Reproducing sensed image data as they are amounts to merely experiencing what a photographer has gone 
is through. For this reason, a technique for generating and presenting an arbitrary scene in real time using the IBR 
technique has been proposed. More specifically, when sensed images are processed as independent ones, and are 
re-arranged in accordance with a viewer's request, the viewer can walkthrough his or her desired moving route at a 
remote place, and can feel a three-dimensional virtual space there. 

[0004] It is effective for searching for and re-constructing a desired image in accordance with the viewer's request 
20 to use the location information of points where the individual images were taken. That is, an image closest to the 
viewer's request is selected from a database, and undergoes proper image interpolation so as to generate and display 
an optimal image. 

[0005] Fig. 1 shows the principle of wide-area walkthrough using sensed images. 

[0006] More specifically, sensed images are prepared for narrow areas 1 , 2, 3, and 4. In order to implement wide- 
ns area walkthrough (e.g., along a route 10 or 11) that allows the viewer to walk across these narrow areas, an image in 
a space between adjacent narrow areas must be obtained by interpolation. When the viewer is currently located at a 
position between the narrow areas 2 and 3, and the space between these areas is obtained by interpolation, specific 
sensed images for the areas 2 and 3 must be obtained by a search on the basis of information associated with the 
current location of the viewer between the areas 2 and 3. In other words, in order to obtain required images by a search 
30 on the basis of the location information of the user, a database of sensed images must be prepared in advance in 
accordance with location data upon image sensing. 

[0007] In order to attain precise interpolation and to smoothly connect the interpolated images and sensed images, 
as the viewer may walk through in the 360° range around him or her, sensed images of the environment must be taken 
by a large number of cameras disposed to point in various directions, and an image database must be built using these 
35 sensed images. 

[0008] In order to attain precise interpolation on the basis of images obtained using a plurality of cameras, the image 
sensing centers of many cameras must agree with each other. However, it is not easy to arrange many cameras in 
such way. 

[0009] To solve this problem, conventionally, a plurality of mirrors are set symmetrically about a given point, and the 
40 mirror surfaces of the individual mirrors are set so that light beams coming from the surrounding portions are reflected 
upward, thereby setting the image sensing centers of the cameras at one point, as shown in Fig. 2. 
[0010] However, in the mirror layout shown in Fig. 2 it is difficult to prevent the photographer (vehicle) from being 
reflected in the mirrors. 

45 SUMMARY OF THE INVENTION 

[0011] The present invention has been made in consideration of the above situation, and has as its object to provide 
an image database forming method for processing sensed images from a plurality of cameras which are separated 
from each other as if they were obtained from cameras which are not separated from each other. With the database, 

50 a high degree of freedom in camera layout can be assured. 

[0012] It is another object of the present invention to provide an image recording apparatus which can assure a high 
degree of freedom in camera layout by setting a camera which senses a Iront image to be separated from a camera 
which senses an image in the direction opposite to the front image while setting cameras on the right and left sides of 
a vehicle to be close to each other, since the vehicle that mounts cameras for. obtaining sensed images travels straight 

55 most of the time. 

[0013] In order to achieve the above objects, according to the present invention, an image recording apparatus for 
recording images sensed by at least two image sensing means attached to a vehicle, is characterized by comprising: 



2 





10 



15 



20 



25 



30 



EP 0 921 375 A1 

first image sensing means which is arranged to have an image sensing direction agreeing with a first direction of 
the vehicle; 

second image sensing means which is arranged at a position separated a known distance from the first image 
sensing means to have an image sensing direction agreeing with a second direction substantially 180° different 
from the first direction; and 

recording means for, when the vehicle travels in the first direction, recording first image data sensed by the first 
image sensing means and second image data sensed by the second image sensing means, a time<Juration later 
corresponding to the known distance, in association with each other. 

[0014] According to a preferred aspect of the present invention, the first image sensing means comprises a plurality 
of cameras, image sensing directions of which are deployed symmetrically about the first direction. 
[0015] According to a preferred aspect of the present invention, straight lines on the image sensing directions of the 
plurality of cameras cross each other in front of the plurality of cameras. 

[0016] According to a preferred aspect of the present invention, the apparatus further comprises: 

third image sensing means which is arranged at a position near the first image sensing means to have an image 
sensing direction agreeing with a third direction different from the first direction; and 

fourth image sensing means which is arranged at a position symmetrical to the third direction about a straight line 
pointing in the first direction. 

[0017] According to a preferred aspect of the present invention, the second image sensing means comprises a 
plurality of cameras, image sensing directions of which point in at least two directions symmetrical about the second 
direction. 

[0018] In order to achieve the above objects, according to the present invention, an image recording apparatus for 
recording images sensed by at least two image sensing means attached to a vehicle, is characterized by comprising: 

first image sensing means which is arranged to have an image sensing direction agreeing with a first direction of 
the vehicle; 

a plurality of cameras which are arranged at positions separated a known distance from the first image sensing 
means to have image sensing directions agreeing with a plurality of directions symmetrical about a second direction 
substantially 180° different from the first direction; 
means for detecting a turn of the vehicle; 

selection means for selecting the camera which points at a larger angle in a counterclockwise or clockwise direction 
from the plurality of cameras depending on whether the vehicle has turned clockwise or counterclockwise from 
3 $ the first direction; and 

recording means for recording first image data sensed by the first image sensing means and second image data 
sensed by the camera selected by the selection means at a timing a duration corresponding to the known distance 
later, in association with each other. 

40 [001 9] According to a preferred aspect of the present invention, the plurality of cameras have two cameras, and the 
image sensing directions of the cameras cross each other on an extending line of the second direction. 
[0020] According to a preferred aspect of the present invention, the plurality of cameras have first to third cameras 
the first camera has an image sensing direction agreeing with the second direction, and image sensing directions of 
the second and third cameras are respectively turned clockwise and counterclockwise to be deployed symmetrically 

45 about the second direction, and 

the selection means selects 

the first camera when the vehicle travels in the first direction, 
the second camera when the vehicle turns counterclockwise, and 
so the third camera when the vehicle turns clockwise. 

[0021] According to a preferred aspect of the present invention, the apparatus further comprises: 

third image sensing means which is arranged at a position near the first image sensing means to have an image 
sensing direction agreeing with a third direction different from the first direction; and 

fourth image sensing means which is arranged at a position symmetrical to the third direction about a straight line 
pointing in the first direction. 
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[0022] In order to achieve the above objects, according to the present invention, an image database apparatus for 
generating a database used for building a three-dimensional image space from image sequences sensed by a plurality 
of image sensing means attached to a vehicle after acquisition of image data, comprises: 

a first reader for reading data from a first image data memory recorded by first image sensing means pointed in a 
first direction; 

a second reader for reading data from a second image memory recorded by second image sensing means which 
is arranged at a position separated a known distance from the first image sensing means to point in a second 
direction substantially 180° different from the first direction; 

a third reader for reading data from a third memory which records a moving position and traveling direction of the 
vehicle; and 

means for associating image data read by the first reader, and image data at a position the known distance later 
of those read by the second reader with each other when traveling directbn data read by the third reader indicates 
that the vehicle is traveling substantially straight. 



[0023] According to a preferred aspect of the present invention, when the image sensing means includes two cameras 
having different directions, 

the associating means associates image data read by the first reader and image data at a position the known 
distance later of those read by the second reader from the camera located at a counterclockwise or clockwise position 
with each other, when the traveling direction data read by the third reader indicates a clockwise or counterclockwise turn. 
[0024] In order to achieve the above objects, according to the present invention, an image recording method of 
recording images sensed by at least two image sensing means attached to a vehicle, is characterized by comprising 
the steps of: 

arranging first image sensing means to have an image sensing direction agreeing with a first direction of the vehicle; 
arranging second image sensing means at a position separated a known distance from the first image sensing 
means to have an image sensing direction agreeing with a second direction substantially 180° different from the 
first direction; and 

recording, when the vehicle travels in the first direction, first image data sensed by the first image sensing means 
and second image data sensed by the second image sensing means at a timing a duration corresponding to the 
known distance later, in association with each other. 

[0025] According to a preferred aspect of the present invention, the first image sensing means comprises a plurality 
of cameras, image sensing directions of which are deployed symmetrically about the first direction. 
[0026] According to a preferred aspect of the present invention, straight lines on the image sensing directions of the 
plurality of cameras cross each other in front of the plurality of cameras. 

[0027] According to a preferred aspect of the present invention, the method further comprises the steps of: 

arranging third image sensing means at a position near the first image sensing means to have an image sensing 
direction agreeing with a third direction different from the first direction; and 

arranging fourth image sensing means at a position symmetrical to the third direction about a straight line pointing 
in the first direction. 

[0028] According to a preferred aspect of the present invention, the second image sensing means comprises a 
plurality of cameras, image sensing directions of which point in at least two directions symmetrical about the second 
direction. 

[0029] In order to achieve the above objects, according to the present invention, an image recording method of 
recording images sensed by at least two image sensing means attached to a vehicle, is characterized by comprising 
the steps of: 



arranging first image sensing means to have an image sensing direction agreeing with a first direction of the vehicle; 
arranging a plurality of cameras at positions separated a known distance from the first image sensing means to 
have image sensing directions agreeing with a plurality of directions symmetrical about a second direction sub- 
stantially 180° different from the first direction; 
detecting a turn of the vehicle; 

selecting the camera which points at a larger angle in a counterclockwise or clockwise direction from the plurality 
of cameras depending on whether the vehicle has turned clockwise or counterclockwise from the first direction; and 
recording first image data sensed by the first image sensing means and second image data sensed by the selected 
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camera at a timing a duration corresponding to the known distance later in association with each other. 

[0030] According to a preferred aspect of the present invention, the plurality of cameras have two cameras, and the 
image sensing directions of the cameras cross each other on an extending line of the second direction. 
[0031] According to a preferred aspect of the present invention, the plurality of cameras have first to third cameras, 
the first camera has an image sensing direction agreeing with the second direction, and image sensing directions of 
the second and third cameras are respectively turned clockwise and counterclockwise to be deployed symmetrically 
about the second direction, and 

the selection step includes the step of selecting 
the first camera when the vehicle travels in the first direction, 
the second camera when the vehicle turns counterclockwise, and 
the third camera when the vehicle turns clockwise. 

[0032] Other features and advantages of the present invention will be apparent from the following description taken 
in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts 
throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 



Fig. 1 is an explanatory view of the principle of wide-area walkthrough to which the present invention can be applied; 
Fig. 2 is a side view showing the schematic arrangement of a camera mounting device arranged in a conventional 
camera layout apparatus; 

Fig. 3 is a block diagram showing the arrangement of a data acquisition system according to an embodiment of 
the present invention; 

Fig. 4 is a diagram showing the final storage locations of various data in the system shown in Fig. 3; 

Fig. 5 is a block diagram showing generation of a time code in a video camera 20 shown in Fig. 3; 

Fig. 6 is a view for explaining the format of data recorded on a video tape 22 of the video camera 20; 

Fig. 7 is a view for explaining the format of data recorded in a hardware disk HD of a PC 30; 

Fig. 8 is a view for explaining the feature of camera layout of the embodiment shown in Fig. 3; 

Fig. 9 is a view for explaining the image sensing azimuths of cameras #1 to #6 in the camera layout shown in Fig. 8; 

Fig. 10 is a flow chart showing the overall processing of the data acquisition system shown in Fig. 3; 

Fig. 11 is a flow chart for explaining some steps in the flow chart in Fig. 10 in more detail; 

Fig. 12 is a flow chart for explaining some other steps in the flow chart in Fig. 10 in more detail; 

Fig. 13 is a flow chart for explaining some other steps in the flow chart in Fig. 10 in more detail; 

Fig. 14 is a view for explaining merits of the camera layout shown in Fig. 8; 

Fig. 1 5 is a chart for explaining the times of image data to be combined in one record of an image database; 
Fig. 1 6 is a block diagram showing the arrangement of a database generation system according to an embodiment 
of the present invention; 

Fig. 17 shows the formats of files other than the formats shown in Figs. 6 and 7; 

Fig. 1 8 is a flow chart for explaining the basic principle of control for database generation of the embodiment shown 
in Fig. 16; 

Fig. 1 9 is a flow chart for explaining some steps in the flow chart in Fig. 1 8 in more detail; 

Fig. 20 is a flow chart for explaining some other steps in the flow chart in Fig. 18 in more detail; 

Fig. 21 is a flow chart showing the control sequence for converting image data from camera #7 into that from a 

camera located at a position 7' in the camera layout shown in Fig. 14; 

Fig. 22 is a view for explaining production of a dead zone and double-image zone upon generation of a panoramic 
image in the embodiment shown in Fig. 16; 

Fig. 23 is a view for explaining the principle of projection onto a cylinder to achieve panoramic image generation; 
Fig. 24 is a flow chart showing the control sequence according to a modification of the control sequence shown in 
Fig. 10; 

Fig. 25 is a flow chart for explaining some steps in the flow chart in Fig. 24 in more detail; 
Fig. 26 is a flow chart for explaining some other steps in the flow chart in Fig. 24 in more detail; 
Fig. 27 is a view for explaining another camera layout; 

. Fig. 28 is a view for explaining acquisition of image data when a vehicle travels straight in the camera layout shown 
in Fig. 27; 
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Fig. 29 is a view for explaining acquisition of image data when a vehicle turns to the left in the camera layout shown 
in Fig. 27; 7 

Fig. 30 is a view for explaining acquisition of image data when a vehicle turns to the right in the camera layout 
shown in Fig. 27; 

Fig. 31 is a view for explaining still another camera layout; 

Fig. 32 is a view for explaining acquisition of image data when a vehicle travels straight in the camera layout shown 
in Fig. 31; 

Fig. 33 is a view for explaining acquisition of image data when a vehicle turns to the left in the camera layout shown 
in Fig. 31; 

Fig. 34 is a view for explaining acquisition of image data when a vehicle turns to the right in the camera layout 
shown in Fig. 31; and 

Fig. 35 is a flow chart showing the generation sequence of an image database according to a third modification. 
DETAILED DESCRIPTION OF THE INVENTION 



[0034] A preferred embodiment to which the present invention is applied will be described hereinafter. This embod- 
iment is directed to a system for acquiring sensed images (to be referred to as an "image acquisition system" herein- 
after), and a system for building an image database to implement wide-area walkthrough from images acquired by the 
acquisition system (to be referred to as an "image database generation system" hereinafter). 

20 [0035] According to the present invention, location information upon image sensing can be added to image data in 
real time. Therefore, according to the present invention, in principle an image database can be generated in real time 
upon image acquisition. However, the data volume of sensed images is huge, and database generation requires data 
edit. Hence, parallel processing of image acquisition and database generation is not always required For this reason 
the system of this embodiment is divided into two systems, i.e., the "image acquisition system" and "image database 

25 generation system". 



30 



<Arrangement of Image Acquisition System> 

[0036] The image acquisition system of this embodiment senses images of a surrounding environment using a plu- 
rality of cameras mounted on a vehicle. Also, on the vehicle, a GPS sensor 40 for detecting the vehicle location a 
posture sensor 41 for detecting the posture of the vehicle body (i.e., camera posture), and an azimuth sensor 42 for 
geomagnetically detecting the azimuth are mounted. 
[0037] Fig. 3 shows the arrangement of the image acquisition system. The image acquisition system is built by 
mounting the respective devices shown in Fig. 3 on a vehicle. 
3S [0038] Seven cameras (20a, 20b,..., 20g) for sensing environment images are mounted. Each camera used for 
example, a video camera DCR-VX1 000 available from Sony Corp. 

[0039] A PC 30 controls the entire image acquisition system, and incorporates a microprocessor Pentium Pro 200 
MHz. The PC 30 and the individual cameras 20 are connected via video/computer interface units 21a 21q (Vbox II 
CI-1100 available from Sony Corp.). 

[0040] The PC 30 and the interface unit 21 g are connected via a known RS232C interface bus, and the interface 
units 21a, 21b,..., 21g are connected by daisy chain (input signal: VISCA IN, output signal: VISCA OUT) As will be 
described later, the PC 30 sends a time code inquiry signal to the individual cameras 20a,..., 20g. In response to the 
inquiry signal, the cameras 20a,. ... 20g output time code information onto a signal line VISCA OUT, and the time code 
information is supplied to the PC 30. 

[0041] Note that the time code is time information written on a magnetic tape that travels at a prescribed speed in 
the video camera at predetermined time intervals, and detection of one time code signal upon reproduction of the 
magnetic tape means an elapse of the predetermined time. That is, based on the number of detected time code signals 
the time required for reproduction from the beginning of the magnetic tape to the current position can be detected. 
[0042] A GPS sensor 40 and a three-axis posture sensor 41 are connected to the PC 30 via RS232C interface buses 
and a geomagnetic azimuth sensor 42 is connected to the PC via an anabg bus. An azimuth signal from the sensor 
42 is A/D-converted by an internal A/D converter board (not shown) inserted in the PC 30. 
[0043] The GPS sensor 40 used in this system used Model 4400 as a kinematic sensor available from Trimble 
Navigation Limited. The kinematic sensor is capable of location measurements with precision of ±3 cm at a sampling 
rate of 5 Hz. a 

ss [0044] The posture sensor 41 used GU-3020 available from DATATECH Corp., which can assure precision of ±0 5° 
for each of the pitch and roll angles, and precision of ±0.9- for the yaw angle. Also, the geomagnetic azimuth sensor 
42 used TMC-2000 available from Tokin Corporation, which can assure precision of ±2°. 

[0045] Note that arithmetic operations of location information based on signals from the GPS sensor 40 may overload 
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the PC 30 since they require high-speed processing. For this reason, as a modification of the data acquisition system 
shown in Fig. 3, a PC dedicated to GPS data arithmetic operations may be added. 
[0046] Fig. 4 shows the recording locations of various data acquired by the system shown in Fig. 3. 
[0047] As is known, the kinematic GPS sensor outputs high-precision time and location data. 
5 [0048] In Fig. 4, a hard disk HD of the PC 30 stores "time codes" from the cameras 20, "time data" and location 
information data" from the sensor 40, "posture information data (POS) n from the posture sensor 41, and "azimuth 
information (AZM)" from the azimuth sensor 42. 

[0049] In Fig. 4, sensed images from the individual cameras (20a,..., 20g) are respectively recorded on video tapes 
22a, 22b,..., 22g. Fig. 5 shows a recording system in each camcorder (20a,..., 20g). As is known, the camcorder 20 
io incorporates a time code generator 23, signal mixer 24, and video tape 22 for data recording. The time code generator 
23 outputs time codes for indexing image frames. That i6, the time codes are recorded on the tape together with image 
data, and specify the frame positions of the recorded image data. 

[0050] In Fig. 5, the time codes from the time code generator 23 and image data from a CCD sensor (not shown) 
are recorded on the tape 22 by the mixer 24 in accordance with a predetermined format. 
is [0051] Fig. 6 shows an example of the recording format of the two types of data (image data and time code data) 
recorded on the tape 22 of the camera 20. More specifically, on the tape, one time code is assigned to each image 
frame. In other words, target image data on a given tape 22 can be searched on that tape 22 by designating the time 
code corresponding to the image data. 

[0052] Fig. 7 shows an example of the recording format of one record of various kinds of information in the hard disk 
20 HD. 

[0053] Upon receiving a time code at an arbitrary timing from an arbitrary camera, the PC 30 writes a set of "time 
data" (received from the sensor 40), "location information" (received from the sensor 40) of the vehicle, "posture infor- 
mation" from the posture sensor 41 , and "azimuth information" from the azimuth sensor 42 at that time in the disk HD 
as a time code/sensor information record file (in the format shown in Fig. 7). More specifically, one time code/sensor 
25 information record consists of the value of a time code TC, a time TIM at that time, number # of the camera that 
generated the time code, vehicle location data LOC, posture data POS, and azimuth data AZM of the vehicle at the 
time of reception of the time code, and the like. 

[0054] As shown in Fig. 4, in the image acquisition system of this embodiment, images are recorded on the magnetic 
tapes, and the location information and time codes are recorded on the hard disk of the PC 30. 

30 [0055] As can be seen from the "time code/sensor information record" shown in Fig. 7, if the value of a certain time 
t x is given, a record having Tl M close to the value of that time ^ can be detected, and the time code TC, vehicle location 
LOC, posture POS, and azimuth AZM can be detected from the record. Using the value of the obtained time code TC, 
the tape 22 can be searched to acquire target image data. In this fashion, arbitrary image data on the tape can be 
combined with the image sensing location and time of that image data. 

35 [0056] The data acquisition system shown in Fig. 3 is directed to generation of reference image data which are used 
for generating an image database suitable for generating a panoramic image. 

[0057] Note that signals that can represent the locations of sensor data and image data correspond to the time code 
in the camera, and time data from the sensor 40 in the sensor signals. In an image database, image data need only 
be finally linked to the location information and posture data of the vehicle. Hence, the file format stored in the data 
40 acquisition system of this embodiment is not limited to those shown in Figs. 6 and 7. For example, two or more hard 
disk drive units may be arranged in the PC 30, and various kinds of most suited sensor information may be stored in 
the individual disks. 

[0058] Acquisition of data images (to be described later) corresponds to a modification of the data acquisition method 
shown in Figs. 6 and 7. 
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<Camera Layout>... Acquisition System 



[0059] Fig. 8 shows the layout of the seven cameras (20a,..., to 20g) on the vehicle in the data acquisition system 
of this embodiment. 

so [0060] In Fig. 8, the upward direction (indicated by an arrow) in Fig. 8 agrees with the direction the vehicle travels. 
As shown in Figs. 8 and 9, the camera 20c (camera #3) and camera 20d (camera #4) are used for sensing an envi- 
ronment in the traveling direction of the vehicle, the camera 20a (camera #1) and camera 20b (camera #2) are used 
for sensing an environment on the left side of the vehicle, and the camera 20e (camera #5) and camera 20f (camera 
#6) are used for sensing an environment on the right side of the vehicle. 

[0061] Note that each hatched region in Fig. 9 indicates a region where the image sensing angles of neighboring 
cameras overlap each other. 

[0062] In Fig. 8, camera #7 (20g) is used for sensing a rear environment. The image sensing center of the camera 
20 is separated a distance r backward from that (indicated by a point T in Fig. 8) of other cameras (20a to 20g) . By 



7 



EP 0 921 375 A1 4£ 

setting the camera 20g on the rear portion of the vehicle, the camera 20g can be prevented from sensing the vehicle 
body due to the presence of many cameras, as has been described earlier in the paragraphs of the prior art. Since the 
camera 20g is set at the rear portion, many cameras can be prevented from overcrowding the central position T, and 
a high degree of freedom in layout of other cameras can be assured. 

5 

Acquisition of Sensed I mages>... Acquisition System 

[0063] The processing flow for image data acquisition by the data acquisition system of this embodiment will be 
explained below. 

70 [0064] Fig. 10 explains the overall sequence of the sensed image data acquisition controlled by the PC 30. 
[0065] Steps S1 00 and S200 are initialization steps. 

[0066] In step S100, a correspondence among time codes TC from all the cameras (20a to 20g) is determined. In 
step S200, a correspondence among the time codes TC and time data TIM from the GPS sensor is determined. In 
steps S100 and S200, any deviation between each time code TC from the camera and the time data TIM from the 
*5 GPS sensor can be detected. 

[0067] In step S300, a correspondence among the time codes TC and information from the individual sensors (posture 
sensor 41 and geomagnetic azimuth sensor 42) is determined. The processing in step S300 is repeated until image 
sensing is completed. 

[0068] Fig. 11 shows the detailed sequence of "time code correspondence determination" processing in step S100. 

20 [0069] More specifically, in step S102, the value of counter k indicating the camera number is set at °2°. The reason 
why "2" is set is that camera #1 (camera 22a) is used as a reference camera for the sake of convenience. In step S1 04, 
one time code from camera #1 is logged. In step S106, one time code from camera #k is logged. In step S10B, it is 
checked if the above-mentioned operations have been repeated a predetermined number of times (N). After the pre- 
determined number of times of operations, N pairs of the time code value of camera #1 and the value of a time code 

25 TC k from arbitrary camera k among cameras #2 to #7 are obtained. By averaging these plurality of (N) pairs of data, 
a correspondence between the time codes of cameras #1 and #k, i.e., "deviation" (difference between TC^ and TC^ 
can be obtained. Such correspondence is obtained for ail the cameras #2 to #7. 

[0070] Fig. 1 2 shows the detailed processing sequence for obtaining a correspondence between the time codes TC 
and time data TIM in step S200 in Fig. 10. 
30 [0071] More specifically, in step S202, time data TIM is obtained from the GPS sensor. In step S204, the value of a 
time code TC A from camera #1 at time data TIM is logged. By repeating the above-mentioned operation several times, 
the difference between the value indicated by the time code TO, from camera #1 and the absolute time (the time TIM 
from the GPS sensor) can be detected. 

[0072] More specifically, from the relationships obtained by the flow charts shown in Figs. 11 and 1 2, when the time 
3S code from certain camera k is TC k , the deviation between TC k and the time code TC 1 from camera #1, i.e., the time 
interval corresponding to the deviation therebetween can be recognized in advance. 

[0073] The operation in step S300 in Fig. 10 is done in synchronism with capturing of surrounding images by the 
seven cameras 22a to 22g. 

[0074] Fig. 1 3 explains step S300 in detail. More specifically, in step S302, a time code TC, is received from reference 
40 camera #1. In step S304, posture data (pitch, roll, and yaw angles) from the posture sensor 41 are stored. In step 
S306, data from the azimuth sensor 42 is acquired. In step S308, based on the obtained time code TC, and 6ensor 
data, one record is recorded in the time code/sensor information file in the hard disk HD. 

<Assure Degree of Freedom in Camera Layoutx. .Acquisition system 

45 , 

[0075] In Fig. 8, camera #7 (20g) senses a rear environment. The image sensing center of the camera 20 is separated 
the distance r from that (indicated by the point T in Fig. 8) of other cameras (20a to 20f). Since the camera 20g is set 
on the rear portion of the vehicle separated by the distance r, the camera 20g can be prevented from sensing. the 
vehicle body, and a high degree of freedom in layout of many cameras can be assured. More specifically, too much 

50 cameras can be prevented from covering the central position T 

[0076] However, in the camera layout shown in Fig. 8, seven images simultaneously sensed by the seven cameras 
are to be processed as a set of image data simultaneously sensed at a single point. As has been described in the 
paragraphs of the prior art, when one interpolated image is generated from sensed images sensed at twodiscontinuous 
locations, if these two image sensing locations are excessively separated, i.e., if the camera 20g is separated farther 

ss from the camera 20a (i.e., large r is set) to assure a high degree of freedom in camera layout, the interpolated image 
and sensed images cannot be smoothly connected. 

[0077] In this system, in order to compensate for the deviation between image data arising from the distance r, as 
shown in Fig. 14, future image data a time duration Dt: 
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Dt = r/v 



(v is the traveling velocity of the vehicle) 



5 



ahead of the time in question (time t), i.e., sensed image data which would have been sensed by camera #7 Dt (= r/ 
v) after time t (camera #7 must have reached the position T by that time), is used together with image data sensed by 
cameras #1 to #6 at the locations at time t (i.e., the locations illustrated in Fig. 14). 

[0078] More specifically, in order to realize a wide-area walkthrough system, an image database as a basis for gen- 
erating an interpolated image from sensed images must be formed by relating sensed images (stores on the magnetic 
tapes 22a to 22g) from cameras #1 to #7 with each other. As shown in Fig. 15, images sensed by cameras #1 to #6 
at actually the same time (e.g., time t-,), i.e., images IM^t-,) to IMep-,) having the same "time data" are combined in 
one record of the database, and as for image data \M? from camera #7 (20g), image data IM^+r/v) at time t 1 + Dt: 



[0079] As described above, in the data acquisition system shown in Fig. 3, the time codes of the cameras have 
20 deviations. Hence, the processing shown in Fig. 1 5 is executed as follows in the data acquisition system shown in Fig. 
3. That is, assume that the deviation of the time code TC 7 from camera #7 with respect to the time code TC 1 from 
camera #1 obtained by the control sequence in Fig. 11 is DTC, i.e., the time code TC 7 from camera #7 advances 
(delays if DTC < 0) by time DTC from the time code TC, from camera #1 . In this case, image data recorded at a future 
location by a time duration: 



(for Dt = r/v) with respect to reference time t of camera #1 is used. 

30 

<Database Generation System> 

[0080] Fig. 16 shows the arrangement of a "database generation system" for generating an image database used 
for synthesizing a panoramic image from the data acquisition system shown in Fig. 3. More specifically, this generation 

3S system is connected to the above-mentioned "data acquisition system- made up of the PC 30, as shown in Fig. 16, 
and generates a database suitable for image interpolation for the purpose of wide-area walkthrough image presentation 
on the basis of video tape files and magnetic disk files (HD in the PC 30) obtained by the "data acquisition system". 
[0081] The data acquisition system shown in Fig. 3 generates two files shown in Figs. 6 and 7 in addition to the files 
on the magnetic tapes 22a to 22g. These two files are linked to each other by time codes TC. More specifically, image 

40 data of arbitrary frames oh the magnetic tapes 22a to 22g are linked to the camera locations, postures, and azimuths 
upon sensing those image data via the time codes TC. Therefore, when images for seven scenes sensed by the seven 
cameras at a certain image sensing time or timing are required, these image data tor the seven scenes can be desirably 
picked up from the seven magnetic tapes using the time code TC. However, the search of the magnetic tape for a 
target image frame is time-intensive. The primary objective of the database generation system of this embodiment is 

45 to exclude images of unnecessary scenes and to move only images of required frames from a low-speed tape to a 
high-speed file (e.g., a magnetic disk file). 

[0082] The system shown in Fig. 3 cannot often generate the time code/sensor information file shown in Fig. 6 due 
to too much data volume to be processed. In other words, the system shown in Fig. 3 divides the time code/sensor 
information file shown in Fig. 6 into different files, and stores time code data TC, time data TIM, and posture data in 

so one file (to be referred to a "first logging file" hereinafter), and location data and time data TIM in another file (to be 
referred to a "second logging file" hereinafter), as shown in, e.g., Fig. 17. Also, image data are recorded on the tape 
as a "third logging file" including a pair of sensed image and time code. Note that the first and second logging files 
have large data volumes since a record is generated every time the output from the GPS sensor 40 is generated and, 
hence, high precision upon image interpolation can be assured. 

55 [0083] The database generation system to be described below converts the three logging files shown in Fig. 17 
generated by the data acquisition system shown in Fig. 3 into a database. 

[0084] In Fig. 16, a database is generated by an image processing computer 50. Video tape files are set in and 
loaded by video decks 60a to 60g. Also, the PC 30 is connected to the computer 50. 
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+ (r/v) 



is combined in that record. 
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DTC + Dt 
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[0085] Fig. 18 shows the overall processing flow of the computer 50. 

[0086] In step S500, the first logging file is transferred from the PC 30 to the computer 50. In step S600, the second 
logging file is transferred to the computer 50. In step S700, the third logging file is transferred from each video deck 
60 to the computer 50. } 
[0087] In step S800, a correspondence between the first and second logging files is determined using time data TIM 
as a key. Fig. 19 shows the processing of this step in more detail. The processing result is stored in a hard disk (not 
shown) in the computer 50. 

[0088] In step S900, a correspondence between the first and third logging files is determined using time code data 
TC as a key. Fig. 20 shows the processing of this step in more detail. The processing result is also stored in a hard 
disk (not shown) in the computer 50. 

[0089] Note that an image from camera #7 of cameras #1 to #7 must proceed to processing different from those for 
images from other cameras, as has been described above in association with Fig. 8. Hence, the correspondence 
determination between the first and second logging files (step S806) and the correspondence determination between 
the first and third logging files (step S906) are modified to the control sequence shown in Fig. 21. 
[0090] More specifically, one record in the first logging file is read out in step S1 000 in Fig. 21 , and time data TIM is 
detected from the readout record in step S1002. 

[0091] As for image data from cameras #1 to #6, the second logging file for each of cameras #1 to #6 is searched 
to find records with TIM having the same value as the detected time data TIM in step S1004. In step S1006, the record 
in the first logging file found in step S1 000 and the records in the second logging files found in step S1004 are combined 
(in effect, information TIM and information LOC are combined). 

[0092] On the other hand, as for camera #7, equation (1) below is calculated using the time data TIM obtained in 
step S1002: 



Then, the second logging file of camera #7 is searched for a record with TIM having the same value as the time data 
TIM given by equation (1) above. Note that f in equation (1) represents the function of converting the time duration r/ 
v into that required in the data acquisition system of this embodiment. In step SI 01 2, the record in the first logging file 
found in step S1000 and the record in the second logging file found in step S1010 are combined. 
[0093] In step S1020, it is checked if the above-mentioned processing is complete for all the records in the first 
logging file. If YES in step S1020, the flow advances to step S1022, and a record obtained by combining those in the 
first and second logging files is written in a database file. 

[0094] In this manner, a database system that allows easy image interpolation for wide-area walkthrough is generated 
in the computer 50. 

<Generation of Panoramic lmage> 

[0095] Fig. 22 is a view for explaining the overlap angle between adjacent field ranges of the six cameras (20a to 
20g) shown in Fig. 8. 

[0096] A general case will be examined below wherein images are synthesized (a panoramic image is generated) 
so that images of an object P located at a distance Lq are smoothly connected upon interpolation of images obtained 
from two cameras (having the same field angle) which are separated by a distance 2d. A triangular region having a 
point P as a vertex in front of the cameras is a dead zone. A point P' is the one in a region doubly sensed by both the 
cameras. If the point P' is a distance L (L ^ Lq) from the camera, the point P' is imaged on the center side of the point 
P in each image by an angle: 



Hence, when the two images are connected, the object P forms double images, and the difference between the imaging 
locations of the double images is: 



TIM = TIM - 




. . . (1) 




- . . (2) 
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The overlap angle between a pair of cameras (20a and 20b, 20c and 20d, or 20e and 201) is set in consideration of 
10 the above difference. 

[0097] Upon calculating the difference between cameras #2 and #3 or cameras #4 and #5, the image sensing centers 
of which are separated by the largest distance, if the joint between images of an object 3 m ahead of the cameras is 
optimized, an object 1 0 m ahead of the cameras forms double images with an angle difference of 6.5° (1 . 1 m in distance), 
and an infinity object forms double images with an angle difference of 9.3°. 
15 [0098] As for cameras #1 to #6, since their image sensing centers are close to each other, image interpolation is 
performed in consideration of double imaging taking the above equation into account. On the other as for image data 
from camera #7 (20g), since image data at time: 

20 l 1 + < r/V > 

is recorded as that at time tj , image interpolation is performed using image data for camera #7 (20g) in the database. 
[0099] In order to obtain a single panoramic image by synthesis, a virtual projection plane must be converted from 
a flat plane to a cylindrical plane. 
25 [0100] Assuming an image (horizontal: 2Z, vertical 2X) sensed by a camera which points in the direction of absolute 
azimuth 0 O and has a horizontal field angle 2-0 w , as shown in Fig. 23. In this image, if an image located at a position 
separated z vertically and x horizontally from the center of the image is projected into a cylindrical projection plane, 
the projection position (0, z') is given by: 



e = tan.il — °tan9 m | + e 0 

• . . (4) 



z = 



are horizontally arranged in line to obtain a single panoramic image. The overlapping portion between adjacent images 
is subjected to blending to provide a continuous change. 

[0101] When panoramic images are formed using the image database including a large number of images sensed 
by the seven cameras, a panoramic image database is formed. The images in the panoramic image database are used 
45 as source images in the walkthrough system of this embodiment. 

Modification of Data Acquisitions. . First Modification 

[0102] The data acquisition shown in Figs. 10 and 11 is effective when the outputs are obtained from the posture 
so sensor 41 irregularly. 

[0103] A modification to be described below is effective when the outputs are obtained from the posture sensor 41 
at predetermined periods and all data are picked up. More specifically, the control sequence shown in Fig. 24 replaces 
that shown in Fig. 1 0. The control in step S1 1 00 is equivalent to that in step S1 00. The control in step S1 200 is equivalent 
to that in step S200. 

ss [0104] In step S1 300, a correspondence between the time code TC and the output POS from the posture sensor 41 
is determined. More specifically, output data from the posture sensor 41 is logged in step S1302 in Fig. 25, and the 
time code from camera #1 is logged in step S1 304. After that, apair of these data are written in the disk. 
[0105] In step S1400 in Fig. 24, a correspondence between the output data POS from the posture sensor 41 and 
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the output AZM from the azimuth sensor 42 is determined. More specifically/the output from the posture sensor 41 is 
logged in step S1402 in Fig. 26, and the output from the azimuth sensor 42 is logged in step S1404. In this way, the 
outputs from the posture sensor 41 and the azimuth sensor 42 are logged as a pair of data. 

[0106] With the method shown in Fig. 24, since no time code is considered in a loop between steps S1500 and 
s S1400, data acquisition can be done at high speed. 

Modification of Camera Layout>... Second Modification 

[0107] In the camera layout shown in Fig. 8, only one camera 20g is set as that for sensing a rear background. In 
io order to cover the rear field of view using a single camera, the camera 20g must be set to have a wide field angle. 
However, when a wide field angle is set, the resolution of the surrounding image portion drops, and images cannot 
often be smoothly connected upon image interpolation. Also, the camera layout shown in Fig. 8 has no problem when 
the vehicle travels straight, but poses problems when the vehicle turns to the right or left. 

[0108] In a modification shown in Fig. 27, rear camera #8 is added. Since two rear cameras are used, the field of 
?5 view of one camera (#7 or #8) can be narrowed. In order to reduce the dead zone, the optical axes of cameras #7 and 
#8 cross each other, as shown in Fig. 27. 

[0109] When the image acquisition system is mounted on the vehicle with the camera layout shown in Fig. 27, if this 
vehicle travels straight, cameras #7 and #8 respectively move to locations 7' and 8' in Fig. 28 a period vN later. More 
specifically, when the vehicle keeps traveling forward, image data from both cameras #7 and #8 several frames (cor- 

20 responding to the period r/v) before are used. ' 

[0110] One of the merits obtained by crossing the optical axes of cameras #7 and #8 each other appears when the 
vehicle turns to the right or left. More specifically, when the vehicle turns to the left, since the vehicle body rotates 
counterclockwise, rear camera #8 reaches a location (8 1 ) shown in Fig. 29. More specifically assume that image data 
from camera #8 at the location 8' is combined with those obtained from cameras #1 to #6 at the locations before the 

2S left turn. In this case, since image data obtained from cameras #1 to #6 when the vehicle body points in a direction 
before the left turn (i.e., straight direction) are combined with that obtained from camera #8 in the straight direction, 
the problem posed by the layout shown in Fig. 8 can be solved. 

[0111] Note that the left or right turn can be detected based on, e.g., the output from the posture sensor 41 . That is, 
the computer 50 reads a file that stores the outputs from the posture sensor 41, and if the turn direction at that time 
30 indicates the right (or left), image data from camera #7 (or #8) is selected. Generally, when the vehicle body turns 
clockwise (counterclockwise), the output from the camera which offsets counterclockwise (clockwise) from the central 
line is used. 

[0112] Fig. 30 shows the case wherein image data from camera #7 is used in case of the right turn. 
[0113] Note that the control sequence of the second embodiment will become apparent from that of a third modifi- 
35 cation to be described below. 



Modification of Camera Layouts. . Third Modification 



[0114] In the third modification (Figs. 31 to 34), camera #8 that faces the rear center of the vehicle is added to the 
second modification. 

[01 16] With this layout, when the vehicle travels straight, image data from central camera #8 is used, as shown in 
Fig. 32; when the vehicle turns to the left, image data from camera #9 set at the right rear side is used, as shown in 
Fig. 33; and when the vehicle turns to the right, image data from camera #7 set at the left rear side is used, as shown 
in Fig. 34. 

[0116] The third modification can connect images more smoothly than in the second modification. 
[0117] Fig. 35 shows the database generation sequence of the third modification. This control sequence is substan- 
tially the same as that in the second modification in that one of image data from the cameras that sense rearview 
images is selected on the basis of the traveling direction of the vehicle (that can be determined based on the posture 
data POS or azimuth data from the sensor 42). The difference from the sequence shown in Fig. 21 is that the right 
turn, straight travel, or left turn is determined on the basis of the posture data POS (or azimuth data from the sensor 
42) in step S201 0, and if the right turn is determined, image data from camera #7 (Fig. 34) is used; if the straight travel 
is determined, image data from camera #8 (Fig. 32) is used; and if the left turn is determined, image data from camera 
#9 (Fig. 33) is used. 

Synchronization of Image Sensing and Database Formation^.. Fourth Modification 

[0118] In the above embodiment, the image database is formed after image sensing on the basis of image data 
recorded on tapes in advance. However, a database may be formed while sensing images. In this case, a large^apacity, 
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high-speed filing device is required. 

[0119] When a database is formed in real time in the embodiment shown in Fig. 14, 27 or 31 , a buffer for delaying 
image data sensed by a camera set at the rear position (camera #7 in the example shown in Fig. 14; cameras #7 and 
#8 in the example shown in Fig. 27; cameras #7 to #9 in the example shown in Fig. 31 ) by the above-mentioned period 
f(r/v) is required. 



<Other Modifications> 



[0120] In the above embodiment, as combinations of data to be stored in the first and second logging files, data TC, 
io TIM, and POS are recorded in the first logging file, and data LOC and TIM are recorded in the second logging file, as 
shown in, e.g., Fig. 17. However, the present invention is not limited to such specific combinations of data shown in 
Fig. 17. 

[0121] More specifically, since image data normally has a large capacity, image data is preferably solely recorded 
on a large-capacity memory such as a magnetic tape. However, data TC, TIM, POS, and LOC may be recorded in a 
*s single high-speed file. 

[0122] According to the embodiment described above, location information associated with each image sensing 
location can be efficiently added to a sensed image sequence obtained by sensing an object using cameras. 
[0123] Also, a database apparatus which builds an image database added with time information so that location 
information associated with each image sensing position can be efficiently added to a sensed image sequence obtained 

20 by sensing an object using cameras can be realized. 

[0124] According to the above embodiment, an image database which processes sensed images from a plurality of 
cameras, which are separated from each other, as if they were taken by cameras which are not separated from each 
other can be formed. With this database, a high degree of freedom in camera layout can be assured. 
[0125] Since the vehicle that mounts cameras for obtaining sensed images travels straight most of the time, the 

25 camera which senses a front image is separated from the camera which senses an image in the direction opposite to 
the front image while setting the cameras on the right and left sides of the vehicle to be close to each other. Hence, 
an image recording apparatus which can assure a high degree of freedom in camera layout can be provided. 
[0126] As many apparently widely different embodiments of the present invention can be made without departing 
from the spirit and scope thereof, it is to be understood that the invention is not limited to the specific embodiments 

30 thereof except as defined in the appended claims. 



Claims 

1. An image recording apparatus for recording images sensed by at least two image sensing means attached to a 
vehicle, comprising: 



40 



45 



SO 



55 



first image sensing means which is arranged to have an image sensing direction agreeing with a first direction 
of the vehicle; and 

second image sensing means which is arranged at a position separated a known distance from said first image 
sensing means to have an image sensing direction agreeing with a second direction substantially 180° different 
from the first direction, 

characterized by recording means for, when the vehicle travels in the first direction, associating first image 
data sensed by said first image sensing means with second image data sensed by said second image sensing 
means a time duration later corresponding to said known distance, and recording said first and second image data. 

2. The apparatus according to claim 1, wherein said first image sensing means comprises a plurality of cameras, 
image sensing directions of which are deployed symmetrically about the first direction. 

3. The apparatus according to claim 2, wherein straight lines on the image sensing directions of said plurality of 
cameras cross each other in front of said plurality of cameras. 

4. The apparatus according to claim 1 , further comprising: 

third image sensing means which is arranged at a position near said first image sensing means to have an 

image sensing direction agreeing with a third direction different from the first direction; and 

fourth image sensing means which is arranged at a position symmetrical to the third direction about a straight 



13 



EP 0 921 375 A1 



line pointing in the first direction. 



5. The apparatus according to claim 1 , wherein said second image sensing means comprises a plurality of cameras, 
image sensing directions of which point in at least two directions symmetrical about the second direction. 

5 

6. An image recording apparatus for recording images sensed by at least two image sensing means attached to a 
vehicle, comprising: 

first image sensing means which is arranged to have an image sensing direction agreeing with a first direction 

10 of the vehicle; 

a plurality of cameras which are arranged at positions separated a known distance from said first image sensing 
means to have image sensing directions agreeing with a plurality of directions symmetrical about a second 
direction substantially 180° different from the first direction; 
means for detecting a turn of the vehicle; 

75 selection means for selecting the camera which points at a larger angle in a counterclockwise or clockwise 

direction from said plurality of cameras depending on whether the vehicle has turned clockwise or counter- 
clockwise from the first direction; and 

recording means for associating first image data sensed by said first image sensing means with second image 
data sensed by the camera selected by said selection means a time duration later corresponding to the known 
20 distance, and recording said first and second image data. 

7. The apparatus according to claim 6, wherein said plurality of cameras have two cameras, and the image sensing 
directions of said cameras cross each other on an extending line of the second direction. 

25 8. The apparatus according to claim 6, wherein said plurality of cameras have first to third cameras, said first camera 
has an image sensing direction agreeing with the second direction, and image sensing directions of said second 
and third cameras are respectively turned clockwise and counterclockwise to be deployed symmetrically about 
the second direction, and 



said selection means selects 

said first camera when the vehicle travels in the first direction, 
said second camera when the vehicle turns counterclockwise, and 
said third camera when the vehicle turns clockwise. 



35 9. The apparatus according to claim 6, further comprising: 

third image sensing means which is arranged at a position near said first image sensing means to have an 
image sensing direction agreeing with a third direction different from the first direction; and 
fourth image sensing means which is arranged at a position symmetrical to the third direction about a straight 
40 line pointing in the first direction. 



10. An image database apparatus for generating a database used for building a three-dimensional image space from 
image sequences sensed by a plurality of image sensing means attached to a vehicle after acquisition of image 
data, comprising: 

45 

a first reader for reading data from a first image data memory recorded by first image sensing means pointed 
in a first direction; 

a second reader for reading data from a second image memory recorded by second image sensing means 
which is arranged at a position separated a known distance from said first image sensing means to point in a 
50 second direction substantially 180° different from the first direction; 

a third reader for reading data from a third memory which records a moving position and traveling direction of 
the vehicle; and 

means for associating image data read by said first reader, and image data at a position the known distance 
later of those read by said second reader with each other when traveling direction data read by said third 
55 reader indicates that the vehicle is traveling substantially straight. 

11. The apparatus according to claim 10, wherein when said image sensing means includes two cameras having 
different directions, 
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said associating means associates image data read by said first reader and image data at a position the 
known distance later of those read by said second reader from the camera located at a counterclockwise or clock- 
wise position with each other, when the traveling direction data read by said third reader indicates a clockwise or 
counterclockwise turn. 

1 2. An image recording method of recording images sensed by at least two image sensing means attached to a vehicle, 
comprising the steps of: - 

arranging first image sensing means to have an image sensing direction agreeing with a first direction ol the 
vehicle; 

arranging second image sensing means at a position separated a known distance from said first image sensing 
means to have an image sensing direction agreeing with a second direction substantially 180° different from 
the first direction; and 

associating, when the vehicle travels in the first direction, first image data sensed by said first image sensing 
means with second image data sensed by said second image sensing means atime duration later correspond- 
ing to the known distance. 

13. The method according to claim 12, wherein said first image sensing means comprises a plurality of cameras, image 
sensing directions of which are deployed symmetrically about the first direction. 

1 4. The method according to claim 1 3, wherein straight lines on the image sensing directions of said plurality of cameras 
cross each other in front of said plurality of cameras. 

15. The method according to claim 12, further comprising the steps of: 

arranging third image sensing means at a position near said first image sensing means to have an image 
sensing direction agreeing with a third direction different from the first direction; and 

arranging fourth image sensing means at a position symmetrical to the third direction about a straight line 
pointing in the first direction. 

16. The method according to claim 1 2, wherein said second image sensing means comprises a plurality of cameras, 
image sensing directions of which point in at least two directions symmetrical about the second direction. 

1 7. An image recording method of recording images sensed by at least two image sensing means attached to a vehicle, 
comprising the steps of: 

arranging first image sensing means to have an image sensing direction agreeing with a first direction of the 
vehicle; 

arranging a plurality of cameras at positions separated a known distance from said first image sensing means 
to have image sensing directions agreeing with a plurality of directions symmetrical about a second direction 
substantially 1 80° different from the first direction; 
detecting a turn of the vehicle; 

selecting the camera which points at a larger angle in a counterclockwise or clockwise direction from said 
plurality of cameras depending on whether the vehicle has turned clockwise or counterclockwise from the first 
direction; and 

recording first image data sensed by said first image sensing means and second image data sensed by the 
selected camera at a timing a duration corresponding to the known distance later in association with each other. 

18. The method according to claim 17, wherein said plurality of cameras have two cameras, and the image sensing 
directions of said cameras cross each other on an extending line of the second direction. 

19. The method according to claim 1 7, wherein said plurality of cameras have first to third cameras, said first camera 
has an image sensing direction agreeing with the second direction, and image sensing directions of said second 
and third cameras are respectively turned clockwise and counterclockwise to be deployed symmetrically about 
the second direction, and 

the selection step includes the step of selecting 

said first camera when the vehicle travels in the first direction, 
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said second camera when the vehicle turns counterclockwise, and 
said third camera when the vehicle turns clockwise. 

20. A recording medium of a computer program which makes a computer execute control tor recording images sensed 
5 by first image sensing means which is arranged to have an image sensing direction agreeing with a first direction 

of the vehicle, and second image sensing means which is arranged at a position separated a known distance from 
said first image sensing means to have an image sensing direction agreeing with a second direction substantially 
180° different from the first direction, said medium recording: 

first program code means for, when the vehicle travels in the first direction, recording first image data sensed 
10 by said first image sensing means and second image data sensed by said second image sensing means in asso- 

ciation with each other. 

21 . A recording medium of a computer program which makes a computer execute control for recording images sensed 
by first image sensing means which is arranged to have an image sensing direction agreeing with a first direction 

is of the vehicle, a plurality of cameras which are arranged at positions separated a known distance from said first 

image sensing means to have image sensing directions agreeing with a plurality of directions symmetrical about 
a second direction substantially 180° different from the first direction, said medium recording: 

first program code means for detecting a turn of the vehicle; 
20 second program code means for selecting the camera which points at a larger angle in a counterclockwise or 

clockwise direction from said plurality of cameras depending on whether the vehicle has turned clockwise or 
counterclockwise from the first direction; and 

third program code means for recording first image data sensed by said first image sensing means and second 
image data sensed by the selected camera at a timing a duration corresponding to the known distance later 
25 in association with each other. 

22. An imaging method for virtual space panoramic synthesis comprising recording plural view images using respective 
plural cameras at multiple view points along an image gathering path, the cameras being oriented to record re- 
spective different views from each view point. 

30 

23. An imaging method as claimed in claim 22 wherein the cameras are mounted on a vehicle at mutually spaced 
locations relative to the vehicle and including the step of recording vehicle attitude and position information as a 
function of time and in synch with the recorded images. 

36 
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